fix(loadbalancer): target per-host proxies on their published http port + re-register services after LB reboot - #107
Merged
Conversation
…ot app_port 3.0.0 moved the LB target construction from Kamal::Commands::Loadbalancer (which used :80) into Configuration::Loadbalancer#deploy_command_args, which built <host>:<app_port>. app_port is how a per-host proxy reaches the app container inside its own docker network - nothing listens on it cross-host, the per-host proxies only publish 80/443 - so every LB deploy timed out with targets that could never become healthy. Use run.http_port (default 80) instead: the LB run config is the same shared proxy/run surface the per-host proxies publish with, so a custom http_port is honoured automatically. Refs #104
A rebooted LB starts from whatever the state volume restored; until 3.0.0 the deploy flow only re-registered services in the LB deploy step that runs AFTER app boot. On the first v3 deploy the drift-digest schema change forces an LB reboot, and when the subsequent deploy step failed (see the target port regression) the LB was stranded with an empty service table - site down. Mirror the per-host proxy reboot instead: once the replacement container is ready, re-register this apps service with the current targets and verify it via kamal-proxy list --json, raising if it is missing. Only this apps registration can be rebuilt from this deploy.yml, so the safety net is gated on the service owner file recording this app; other apps sharing the LB still ride on the state-volume restore. Target collection moves to Configuration::Loadbalancer#target_hosts so the deploy step and the reboot re-registration draw from one source of truth. Refs #104
mhenrixon
force-pushed
the
fix/issue-104-lb-target-port
branch
from
August 3, 2026 20:46
e296f24 to
9926121
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #104
What
Two fixes for the 3.0.0 loadbalancer deploy regression:
Target port —
Kamal::Configuration::Loadbalancer#deploy_command_argsbuilt targets as<host>:<app_port>.app_portis how a per-host proxy reaches the app container inside its own docker network; nothing listens on it cross-host — the per-host proxies only publish 80/443 — so every LB deploy timed out ("target failed to become healthy"). Targets now userun.http_port(default 80): the LB's run config is the same sharedproxy/runsurface the per-host proxies publish with (publishmaps hosthttp_port→ container 80), so a customproxy.run.http_portis honoured automatically.Reboot re-registration safety net — the LB reboot path (drift-detected on boot, or
kamal proxy reboot) previously relied entirely on the state-volume restore plus the LB deploy step that runs after app boot. On the first v3 deploy the drift-digest schema change forces an LB reboot, and when the subsequent deploy step failed (bug 1) the LB was stranded with an empty service table — site down.Kamal::Cli::Proxy::LoadbalancerRebootnow mirrors the per-host proxy reboot: wait until the replacement container answerskamal-proxy list(30s deadline), re-register this app's service with the current targets, and verify it vialist --jsonexact key membership, raisingKamal::Cli::BootErrorif missing.Supporting changes:
Kamal::Configuration::Loadbalancer#target_hosts— target collection (hosts of every proxy-running role) extracted fromKamal::Cli::Proxy#loadbalancerso the deploy step and the reboot re-registration draw from one source of truth. Same semantics (order, no dedup).Kamal::Commands::Loadbalancer#listgained ajson:param, mirroringKamal::Commands::Proxy#list.Why re-register only this app's service
The issue suggests re-registering everything recorded under
.kamal/loadbalancer/services/*, but those files record owner tokens, not deploy commands — another app's targets and options cannot be rebuilt from this app's deploy.yml. Other apps' routes survive through thekamal-loadbalancer-configstate volume (re-verified by--recheck-targets-on-restore); this app's registration is additionally re-registered fresh and verified, gated on the owner file recording this app. No owner record (never deployed) or a foreign owner skips the step silently.Test plan
test/commands/loadbalancer_test.rbapp_port: 3000→ targets stay:80)proxy.run.http_port: 8080→ targets:8080list --json,target_hoststest/cli/proxy_test.rbkamal-proxy deploy app --target="1.1.1.1:80,1.1.1.2:80"exec on the LB)list --jsonafter re-registrationbundle exec rubocop --parallel— clean (218 files)Deviations & judgment calls
run.http_port(default 80), not a hardcoded 80:KAMAL.loadbalancer_configis built from the root proxy config, so itsrunis the same proxy/run block the per-host proxies publish with — a customproxy.run.http_portis respected automatically, and apps sharing an LB already must agree on proxy/run (run-config digest claim), so the value cannot diverge per host..kamal/loadbalancer/services/*as the issue literally suggests: those files record owner tokens only — another app's deploy options/targets cannot be rebuilt from this app's deploy.yml. Other apps' routes survive via the state volume; ours is additionally re-registered fresh so a later app-deploy failure can't strand the LB without it.wait_until_ready(retrykamal-proxy listfor up to 30s) before re-registration, mirroringKamal::Cli::Proxy::Reboot::READY_TIMEOUT— the old code listed services immediately afterdocker runand could race a slow container start.Kamal::Configuration::Loadbalancer#target_hosts; preserved exact old semantics (no uniq/dedup, same ordering).proxy.run.publish: falsemakes the per-host proxies unreachable from a dedicated LB host entirely; that topology error pre-dates this fix and is not newly validated here.lib/kamal/configuration/docs/proxy.yml) never claimed the LB targets app_port, so no doc change needed.